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Abstract 


This  report  describes  the  inaugural  Measuring  What  Matters  Workshop  conducted  in  November 
2014,  and  the  team’s  experiences  in  planning  and  executing  the  workshop  and  identifying  im¬ 
provements  for  future  offerings.  The  Measuring  What  Matters  Workshop  introduces  the  Goal- 
Question-Indicator-Metric  (GQIM)  approach  that  enables  users  to  derive  meaningful  metrics  for 
managing  cybersecurity  risks  from  strategic  and  business  objectives.  This  approach  helps  ensure 
that  organizational  leaders  have  better  information  to  make  decisions,  take  action,  and  change  be¬ 
haviors. 
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1 


Introduction 


1.1  Purpose 

This  report  describes  the  inaugural  Measuring  What  Matters  Workshop  conducted  in  November 
2014,  and  the  experience  of  the  team — staff  of  the  CERT  Division  of  the  Carnegie  Mellon  Uni¬ 
versity  Software  Engineering  Institute  (SEI) — in  planning  and  executing  the  workshop  and  identi¬ 
fying  improvements  for  future  offerings.  The  Measuring  What  Matters  Workshop  introduces  a 
measurement  approach  that  enables  users  to  derive  meaningful  metrics  for  managing  cybersecu¬ 
rity  risks  from  strategic  and  business  objectives.  This  approach  helps  ensure  that  organizational 
leaders  have  better  information  to  make  decisions,  take  action,  and  change  behaviors.  It  also  helps 
ensure  that  planning,  budgeting,  and  the  allocation  of  resources  are  focused  on  monitoring  what 
matters  most  to  the  organization. 

1.2  Workshop  Approach 

The  Measuring  What  Matters  Workshop  uses  a  derivative  of  the  Goal-Question-Indicator-Metric 
(GQIM)  approach  [Park  1996]  to  derive  example  metrics  from  a  stated  strategic  or  business  ob¬ 
jective.  We  first  demonstrate  the  approach  using  a  simple  objective:  teaching  a  child  to  properly 
brush  his/her  teeth.  Next,  we  demonstrate  the  approach  using  a  cybersecurity  incident  manage¬ 
ment  example.  We  then  present  a  detailed  description  of  a  security  incident  experienced  by  Forbes 
in  2014  and  demonstrate  how  metrics  are  derived  from  a  set  of  objectives  designed  to  ensure  that 
such  incidents  do  not  recur.  Last,  we  ask  participants  to  select  a  business  objective  from  their  own 
organizations  and  apply  the  GQIM  process  to  derive  meaningful  metrics  to  take  home.  After  com¬ 
pleting  the  workshop,  participants  should  understand  the  elements  of  a  measurement  program  and 
how  to  get  one  started. 

As  a  result  of  the  workshop,  participants  should  be  able  to 

•  demonstrate  the  business  value  of  each  metric  (and  thus  justify  the  cost  for  collecting  and  re¬ 
porting  the  metric) 

•  defend  meaningful  metrics  in  comparison  to  others 

•  add  metrics,  update  metrics,  and  retire  metrics  as  business  objectives  change 

•  use  metrics  to  inform  business  decisions,  take  appropriate  action,  and  change  behaviors 
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2  Workshop  Overview 


This  section  provides  a  brief  description  of  the  background  that  motivated  our  development  of  the 
workshop,  workshop  participants,  and  the  eight  topics  that  composed  the  workshop. 

2.1  Background 

One  of  the  common  mistakes  that  organizations  make  when  they  embark  on  a  measurement  pro¬ 
gram  is  to  begin  collecting  whatever  data  is  available  and  define  and  report  metrics  based  on  that 
data.  Often  they  are  dissatisfied  with  the  results,  which  may  lack  information  or  a  clear  direction 
for  action.  Foundational  elements  that  are  typically  missing  include 

•  the  identification  of  key  stakeholders  and  audiences — the  customers  and  users  of  the  results 
generated  by  the  measurement  program 

•  the  identification  of  strategic  and  business  objectives  that  the  measurement  program  is  in¬ 
tended  to  support 

•  the  development  of  candidate  questions  that  stakeholders  are  seeking  to  answer  based  on  the 
resulting  metrics 

It  is  critical  to  measure  the  right  things  in  order  to  make  informed  decisions,  take  the  appropriate 
actions,  and  change  behaviors.  But  how  do  senior  leaders  and  managers  figure  out  what  those 
right  things  are? 

Public  and  private  organizations  today  often  base  cybersecurity  risk  management  decisions  on 
fear,  uncertainty,  and  doubt;  the  latest  attack  reported  in  the  press;  compliance  mandates  such  as 
the  Health  Insurance  Portability  and  Accountability  Act,  Federal  Information  Security  Manage¬ 
ment  Act,  Sarbanes-Oxley  Act,  and  Payment  Card  Industry  Data  Security  Standard;  and  security 
risk  frameworks  that  typically  have  little  to  do  with  the  way  the  rest  of  the  organization  measures 
risk  and  prioritizes  operational  risk  management  activities. 

Chief  financial  officers,  enterprise  risk  management  officers,  internal  audit  directors,  and  chief  in¬ 
formation  security  officers  need  cybersecurity  risk  management  approaches  that  align  with  and 
support  the  achievement  of  business  objectives. 

A  measurement  approach  tied  to  strategic  and  business  objectives  ensures  that  planning,  budget¬ 
ing,  and  allocating  operational  resources  are  focused  on  what  matters  most  to  the  organization.  In 
addition,  a  shift  to  such  an  approach  may  help  to  identify  metrics  that  are  expensive  to  collect  and 
may  not  be  worth  the  investment. 

The  report  extends  and  applies  the  operational  resilience  measurement  concepts  described  in  the 
work  of  Allen  and  colleagues  [Allen  2010,2011a,  2011b], 

The  workshop  was  presented  as  a  one-day  offering  at  the  ISACA  Information  Security  and  Risk 
Management  Conference  in  Las  Vegas,  Nevada,  on  November  18,  2014. 
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2.2  Participants 


Forty-six  U.S.  and  international  participants  from  a  range  of  market  sectors  registered  for  the 
workshop.  Of  those,  34  completed  workshop  evaluation  forms  on  which  they  shared  more  de¬ 
tailed  information  on  their  purpose  for  attending,  market  sector,  position,  and  years  of  experience. 
In  addition,  20  participants  responded  to  our  request  to  provide  example  strategic  and  business  ob¬ 
jectives  in  advance  as  part  of  the  workshop  pre-work.  Those  participants  provided  additional 
background  on  their  organizational  affiliation  and  roles.  This  section  provides  an  aggregate  de¬ 
scription  of  this  information. 

The  34  participants  who  completed  workshop  evaluation  forms  stated  the  following  reasons  for 
attending  (participants  could  select  multiple  reasons): 

•  Gain  awareness:  4 

•  Improve  skills:  18 

•  Implement  concepts:  17 

•  Teach  others:  2 

Participants  had  an  average  of  15-20  years  of  experience,  with  a  minimum  of  2  years  and  a  maxi¬ 
mum  of  32  years. 

Participants  represented  the  following  market  sectors: 

•  Financial  services 

•  Health 

•  U.S.  federal  civilian  agency 

•  U.S.  Department  of  Defense 

•  Telecommunications 

•  Retail 

•  IT  and  security  consulting 

•  (Unspecified)  international  industry 

Participants  reported  holding  the  following  job  titles: 

•  VP,  operational  risk 

•  Director,  client  services 

•  Director,  technology  risk 

•  Director,  risk  assurance 

•  Chief  information  officer 

•  Information  security  officer 

•  Compliance  and  security  officer;  IT  compliance 

•  Manager  of  (IT/information)  governance,  security,  risk,  and  compliance 

•  General  manager  and  program  manager 

•  Security  professional,  security  engineer,  security  architect,  and  network  engineer 
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•  (Security)  risk  analyst  and  IT  risk  management 

•  IT  audit 

The  objectives  provided  by  20  participants  in  advance  greatly  influenced  our  discussions  during 
Topic  2  of  the  workshop.  See  Section  3.2  for  a  description  of  these  objectives  and  Sections  3.3- 
3.7  for  a  description  of  how  they  were  used  during  the  GQIM  process  exercises. 

2.3  Agenda 

The  workshop  was  organized  into  eight  topics  as  shown  in  Figure  1. 


Workshop  agenda 

Topic  1 

Set  context 

Topic  2 

Select  objectives 

Topic  3 

Goal-Question-Indicator-Metric  (GQIM) 
overview 

Topic  4 

Objectives  to  goals 

Topic  5 

Goals  to  questions 

Topic  6 

Questions  to  indicators 

Topic  7 

Indicators  to  metrics 

Topic  8 

The  big  picture:  putting  it  all  in  context 

Softwn  Enqrnf  mq  Intfrtuto  ( «rnrgir  \|rlln«i  1  nHrrtfcy 

Figure  1:  Workshop  Agenda 
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3  Workshop  Topics 


This  section  provides  a  description  of  each  workshop  topic,  example  scenarios,  and  exercises  per¬ 
formed  by  participants. 

3.1  Topic  1:  Set  Context 

3.1.1  Workshop  Expectations 

The  facilitation  team  began  the  workshop  by  establishing  a  baseline  for  participant  expectations 
and  desired  outcomes.  The  lead  facilitator  used  the  workshop  abstract  and  a  set  of  learning  objec¬ 
tives  to  describe  the  concepts  we  planned  to  discuss  during  the  workshop. 


Workshop  abstract 


It  is  critical  to  measure  the  right  things  in  order  to  make  better-informed  decisions,  take  the 
appropriate  actions,  and  change  behaviors.  But  how  do  senior  leaders  and  managers  figure 
out  what  those  right  things  are? 

Public  and  private  organizations  today  often  base  cyber  risk  management  decisions  on  fear, 
uncertainty,  and  doubt  (FUD)  and  the  latest  attack;  compliance  mandates  such  as  HIPAA, 
FISMA,  SOX  and  PCI;  and  security  risk  frameworks  that  typically  have  little  to  do  with  the  way 
the  rest  of  the  organization  measures  risk  and  prioritizes  operational  risk  management 
activities. 

CFOs,  Enterprise  Risk  Management  Officers,  Internal  Audit  Directors,  and  CISOs  need 
information  risk  management  approaches  that  align  with  business  objectives. 

A  measurement  approach  tied  to  strategic  and  business  objectives  ensures  that  planning, 
budgeting,  and  the  allocation  of  operational  resources  are  focused  on  what  matters  most  to 
the  organization.  In  addition,  a  shift  to  such  an  approach  helps  to  identify  metrics  that  are 
expensive  to  collect  and  may  not  be  worth  the  investment. 

Participants  in  this  workshop  will  use  their  real  world  business  objectives  to  develop 
applicable  goals,  questions,  indicators,  and  actionable  metrics  that  they  can  take  back  to  their 
organization  to  improve  their  ability  to  manage  operational  risk  and  resilience. 


CERT  Software  Engineering  Institute  <  .im-tir  \W41nn  I  uKrr>k> 


Figure  2:  Workshop  Abstract 
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Learning  objectives 


1.  Participants  are  expected  to  provide  one  or  more  business 
objectives  from  which  metrics  will  be  derived.  Based  on  a 
defined  business  objective,  select  a  few  essential  goals  that  are 
required  to  achieve  this  objective. 

2.  Formulate  one  or  more  questions  for  each  goal  in  learning 
objective  1 .  The  answers  to  these  questions  help  determine  the 
extent  to  which  the  goal  is  being  achieved. 

3.  Identify  one  or  more  indicators  for  each  question.  An  indicator  is 
data  and  information  that  are  used  to  answer  each  question. 

4.  Using  indicators,  determine  what  number,  percentage,  mean  or 
other  metric  can  help  answer  each  question. 

5.  Understand  the  elements  of  a  measurement  program  and  how 
to  get  one  started. 


cerT  Software  Engineering  Institute  i  nmuir  \lrflnn  l  lahmfet 


Figure  3:  Learning  Objectives 

The  lead  facilitator  asked  participants  to  provide  their  personal  expectations  of  the  workshop.  We 
received  the  following  inputs: 

•  Ways  to  objectively  measure 

•  What  to  do  with  all  the  data — get  to  the  “So  what?” 

•  How  to  make  the  value  translation 

•  Ownership  and  accountability 

•  CIO  to  be  more  transparent  to  the  CEO 

•  How  to  present  metrics  in  an  effective  way 

•  How  to  measure  things  that  are  disparate/ways  to  normalize 

•  Reporting  to  the  board  of  directors,  compliance  committee 

•  Derive  from  metrics  program  if  current  risks  are  appropriate 

3.1.2  Operational  Risk  Management 

Next,  the  facilitator  led  a  discussion  on  operational  risks  and  the  organizational  challenges  faced 
when  managing  these  risks.  For  the  purposes  of  the  workshop,  we  established  the  following  defi¬ 
nition: 

Operational  Risk:  A  form  of  risk  emanating  from  day-to-day  business  operations ;  the  poten¬ 
tial  failure  to  achieve  mission  objectives;  typically  categorized  as  inadvertent  or  deliberate 
actions  of people,  systems  and  technology  failures,  failed  internal  processes,  or  external 
events. 

We  introduced  participants  to  the  concept  of  Operational  Risk  Management  (ORM)  and  its  rela¬ 
tionship  to  security,  business  continuity,  and  IT  operations.  Each  of  these  three  activities  is  essen¬ 
tial  for  managing  operational  risk.  The  facilitator  discussed  our  observations  of  the  current  state  of 
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risk  management:  Risk  is  managed  in  silos  and  generally  in  an  ad  hoc  manner,  and  risk  assump¬ 
tions  are  not  well  understood  or  effectively  communicated  across  organizations. 

The  facilitator  then  introduced  the  idea  that  risk  management  should  ultimately  drive  decision 
making.  When  an  organization’s  risk  management  activities  do  not  drive  decisions,  the  organiza¬ 
tion’s  leaders  must  consider  why  the  organization  is  performing  these  activities.  The  GQIM  pro¬ 
cess  helps  organizations  ensure  that  their  measurement  systems  are  aligned  with  their  strategic 
business  objectives;  the  process  creates  a  holistic  approach  for  managing  operational  resilience. 
An  organization’s  resilience  capability  increases  by  managing  both  sides  of  the  risk  equation 
(condition  and  consequence)  in  alignment  with  business  drivers  and  full  knowledge  of  costs. 

The  facilitator  then  asked  participants  to  provide  their  thoughts  on  the  following  questions: 

•  What  current  barriers  do  you  face  in  establishing,  managing,  and/or  executing  a  measure¬ 
ment  program? 

•  What  challenges  do  you  face  in  identifying  meaningful  metrics  within  your  organization? 
The  participants  identified  the  following  set  of  challenges: 

•  Business  velocity;  need  to  slow  down  to  identify  risks,  ownership,  and  accountability 

•  Having  knowledgeable  risk  management  resources 

•  Lack  of  understanding  of  the  need  to  measure  anything — measure  it  all 

•  Showing  future  value  of  investments  if  everything  is  going  well 

•  Complexity  of  systems,  processes,  and  knowing  where  to  start 

•  Business  units  discount  metrics 

•  Measuring  ROI,  especially  with  insufficient  data 

•  Have  to  quantify  what  you  have  prevented/avoided 

•  Decision  rights  and  conflict  resolution 

Participants  were  asked  what,  if  anything,  they  were  currently  doing  to  overcome  these  chal¬ 
lenges.  They  identified  the  following  actions: 

•  Shifting  accountability 

•  Sponsorship  at  a  leadership  level 

•  Cultural  change 

•  Empowerment  at  lower  levels  in  the  organization 

•  Helping  business  units  to  own  and  take  action 


3.1.3  Measurement 

Next,  the  facilitator  led  a  discussion  on  measurement,  explaining  that  leaders  of  organizations  of¬ 
ten  ask,  “How  secure  is  our  organization?”  That  is, 


How  secure  are  we  compared  to  our  competition? 
Are  we  managing  our  risks  well? 
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•  Do  we  need  to  spend  more  money  on  security  or  risk  management?  If  so,  on  what? 

•  What  are  the  public  relations  and  legal  impacts  of  a  data  breach? 

To  properly  answer  these  questions,  leaders  must  also  answer  the  following  questions: 

•  What  should  we  be  measuring  to  determine  if  we  are  meeting  our  performance  objectives  for 
security? 

Do  we  know  what  these  performance  objectives  are? 

Do  our  performance  objectives  reflect  today’s  realities? 

•  What  is  the  business  value  of  being  more  secure? 

Of  a  specific  security  investment? 

•  So  what?  If  we  had  this  metric  [Hubbard  2010], 

What  decisions  would  it  inform? 

What  actions  would  we  take  based  on  it? 

What  behaviors  would  it  affect? 

What  would  improvement  look  like? 

What  would  its  value  be  in  comparison  to  other  metrics? 

3.1.4  GQIM  Overview 

The  lead  facilitator  described  the  purpose  of  the  GQIM  process  and  provided  a  brief  overview  of 
the  process  steps. 


Purpose 

Use  a  defined,  repeatable  process  to  derive 
meaningful  metrics  that  directly  support  the 
achievement  of  business  objectives 

As  a  result,  be  able  to: 

•  demonstrate  the  business  value  of  each  metric  (and  thus 
justify  the  cost  for  its  collection  and  reporting) 

•  defend  such  metrics  in  comparison  to  others 

•  add  metrics,  update  metrics,  and  retire  metrics  as 
business  objectives  change 

•  ultimately,  inform  business  decisions,  take  appropriate 
action,  and  change  behaviors 

CER^  Software  Engineering  Institute  <  iinwxir  \Mkm  l  iiKrr>*i 

Figure  4:  GQIM  Process  Purpose 
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Figure  5:  GQIM  Process  Steps 


3.2  Topic  2:  Select  Objectives 

As  part  of  the  pre-work  for  the  workshop,  we  asked  participants  to  identify  two  or  three  strategic 
(enterprise)  or  business  (unit-level)  objectives  to  which  they  would  apply  the  GQIM  process.  We 
encouraged  them  to  use  the  SMART(ER)  criteria  shown  in  Figure  6  in  defining  their  objectives. 

SMART(ER)  criteria  for  objectives 

S:  Specific 
M:  Measurable 
A:  Achievable 

R:  Relevant  (Results-based;  Realistic) 

T:  Time-bound 
E:  Evaluated 
R:  Reviewed 


Software  Engineering  Institute  I  imytr  IIHImiI  iilmqii 

Figure  6:  SMART(ER)  Criteria  for  Identifying  Objectives 

About  half  of  those  registered  for  the  workshop  provided  objectives  in  advance.  The  facilitation 
team  integrated  the  participants’  objectives  with  example  objectives  the  team  had  developed  dur¬ 
ing  workshop  preparation.  We  used  the  resulting  objectives  to  group  participants  with  similar  ob- 
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jectives  at  the  same  table  so  they  could  benefit  by  working  with  one  another  on  objectives  of  com¬ 
mon  interest.  Working  with  a  group  of  peers  with  similar  objectives  increases  the  likelihood  of 
information  sharing,  shared  insights,  and  focus  on  the  GQIM  process  (vs.  interpretations  of  any 
specific  objective). 

During  the  Topic  2  discussion,  participants  were  instructed  to  identify  a  business  objective  (if  they 
had  not  done  so  in  advance)  or  modify  an  existing  one  that  they  would  use  to  apply  the  GQIM 
process. 

Based  on  the  facilitation  team’s  objectives  and  those  provided  in  advance  by  participants,  we 
identified  the  candidate  objectives  in  Figure  7  to  be  used  during  the  workshop. 


Candidate  Objectives 

1.  Protect  customer  information 

2.  Keep  software  assets  up-to-date 

3.  User  awareness  of  cybersecurity  threats 

4.  Reliance  on  external  parties 

5.  Mitigate  the  risks  of  disruptive  events/incidents 

6.  Forbes  case  study  (social  engineering/phishing) 


CERT  ^  Software  Engineering  Institute  ( dmrclr  tlrltm  I  nl>mki 

Figure  1:  Candidate  Objectives  for  Table  Assignments 


Each  participant  selected  one  of  the  objectives  and  we  seated  participants  together  according  to 
the  objectives  they  chose.  Several  of  the  objectives  were  so  popular  that  we  needed  two  tables  to 
accommodate  the  participants  who  chose  them. 


A  few  participants  were  consultants  working  with  multiple  organizations  and  did  not  have  a  spe¬ 
cific  business  objective  of  their  own.  We  suggested  that  these  participants  use  the  Forbes  scenario 
(provided  in  the  pre-work)  as  their  objective  for  applying  GQIM.  The  Forbes  scenario  is  described 
in  more  detail  in  Section  3.3. 


The  facilitation  team  also  completed  a  detailed  GQIM  analysis  for  several  of  the  candidate 
objectives  prior  to  the  workshop.  The  team  used  these  analyses  to  facilitate  exercises  and  provided 
them  to  participants  at  the  end  of  the  workshop  for  further  study. 
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3.3  Topic  3:  GQIM  Overview  and  Scenarios 


The  purpose  of  Topic  3  was  to  provide  an  overview  of  the  GQIM  process  and  illustrate  it  using 
several  scenarios.  To  introduce  the  GQIM  process,  we  developed  and  presented  goals,  questions, 
indicators,  and  metrics  based  on  an  easily  understood  objective:  Ensure  your  child’s  teeth  are 
healthy. 

Next,  we  used  a  more  relevant  cybersecurity  risk-related  objective  to  further  illustrate  the  GQIM 
process:  Mitigate  the  risks  of  business  disruption  and  loss  resulting  from  cybersecurity  incidents 
(with  impact  threshold  >  W). 

Throughout  Topics  4,  5,  6,  and  7,  we  used  objectives  derived  from  a  real-world  incident  referred 
to  as  the  “Forbes  scenario.”  The  Syrian  Electronic  Army  used  a  social  engineering  phishing  attack 
to  access  and  compromise  Forbes.com  and  its  companion  publishing  application  in  February 
2014.  The  attack  and  its  impact  are  summarized  as  follows  and  are  further  detailed  in  several  ac¬ 
counts  [Ducklin  2014,  DVorkin  2014,  Greenberg  2014]: 

On  13  Feb  2014,  a  single,  successful  spear  phishing  email  set  in  motion  a  very  public  compro¬ 
mise  ofForbes.com. 

The  Syrian  Electronic  Army  leveraged  the  variety  of  social  media  accounts  that  the  Forbes 
staffers  and  contributors  have  to  leap-frog  from  their  email  accounts  to  the  publication ’s  blog 
and  social  media  platforms. 

All  passwords  across  multiple  platforms  were  forced  to  be  reset  as  a  security  measure  and 
Forbes.com  and  its  WordPress  platform  were  taken  offline  several  times  over  2  days. 

Forbes  has  focused  on  building  unique  content  and  a  publishing  model  for  the  social  media  era 
in  an  open  and  secure  platform. 

We  provided  the  following  objectives  as  a  starting  point  to  apply  the  GQIM  process  to  this  sce¬ 
nario: 

•  Strategic  objective:  Provide  a  content  and  publishing  model  for  the  era  of  social  media  that  is 
both  open  and  secure. 

•  Business  objective:  Increase  user  awareness  of  potential  threats  and  the  appropriate  re¬ 
sponses  to  social  engineering  and  phishing  tactics. 

•  Business  objective:  Improve  the  public’s  and  users’  confidence  in  the  ability  ofForbes.com 
to  operate  securely  and  to  protect  user  privacy. 

The  following  sections  describe  questions  to  ask  at  each  step  in  the  GQIM  process  and  how  to  ap¬ 
ply  each  step  to  these  three  scenarios — ensuring  healthy  teeth,  incident  management,  and  the 
Forbes  scenario — to  generate  example  goals,  questions,  indicators,  and  metrics. 

Using  these  examples  as  references,  participants  then  used  their  selected  objective  to  walk 
through  the  GQIM  process.  Some  of  their  experiences  are  described  in  Section  4. 

3.4  Topic  4:  Objectives  to  Goals 

To  derive  goals  from  objectives,  it  is  useful  to  answer  the  following  questions: 
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•  What  are  meaningful  actions  to  take  to  achieve  the  objective? 

•  Which  actions  are  most  important  (high  leverage,  high  payoff)? 

•  If  I  achieve  this  goal,  will  I  be  able  to  demonstrate  substantive  progress  in  achieving  the  ob- 
jective? 


Table  1,  Table  2,  and  Table  3  provide  example  goals  for  selected  objectives  from  the  three  scenar¬ 
ios  described  in  Section  3.3. 

Table  1:  Objectives  to  Goals — Ensuring  Healthy  Teeth 


Objective 

Goal 

Ensure  you  child’s  teeth  are  healthy. 

G1 :  Ensure  your  child  has  everything  needed  to  brush 
his/her  teeth. 

G2:  Ensure  your  child  is  brushing  his/her  teeth  at  least 
twice  daily. 

Table  2:  Objectives  to  Goals — Incident  Management 

Objective 

Goal 

Mitigate  the  risks  of  business  disruption  and  loss  result¬ 
ing  from  cybersecurity  incidents  (with  impact  threshold 
>  M). 

Operate  a  cybersecurity  incident  center  that  detects,  re¬ 
sponds  to,  and  reports  security  incidents  in  accordance 
with  established  standards  and  guidelines. 

Table  3:  Objectives  to  Goals — Forbes  Scenario 

Objective 

Goal 

Increase  user  awareness  on  potential  threats  and  the  Ensure  users  whose  accounts  are  compromised  do  not 
appropriate  responses  to  social  engineering  and  phish-  succumb  to  the  same  attack(s)  again  (using  random 
ing  tactics.  testing  for  one  year  following  a  compromise). 


3.5  Topic  5:  Goals  to  Questions 

To  derive  questions  from  goals,  it  is  useful  to  answer  the  following  questions: 

•  What  are  meaningful  questions  to  answer  to  determine  if  the  goal  is  being  achieved? 

•  Which  questions  are  most  important? 

•  If  I  answer  this  question,  will  I  be  able  to  demonstrate  substantive  progress  in  achieving  the 
goal? 

Useful  questions  are  often  in  the  fonn  of 

•  What  is  the  process  for  x?  (This  question  provides  a  more  informative  answer  than  “How 
does  the  organization  do  x?”) 

•  How  effective  is  x? 

Table  4,  Table  5,  and  Table  6  provide  example  questions  for  selected  goals  by  scenario. 
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Table  4:  Goals  to  Questions — Ensuring  Healthy  Teeth 
Goal  Question 


G1 :  Ensure  your  child  has  everything  needed  to  brush 
his/her  teeth. 

G2:  Ensure  your  child  is  brushing  his/her  teeth  at  least 
twice  daily. 


Q1:  Does  the  child  have  a  good  toothbrush? 

Q2:  Does  the  child  know  how  to  brush  properly? 

Q1 :  Does  your  child  show  you  his/her  clean  teeth? 


Table  5:  Goals  to  Questions — Incident  Management 


Goal 


Question 


G1 :  Operate  a  cybersecurity  incident  center  that  de-  Q1 :  What  is  the  process  by  which  suspicious  events  are 
tects,  responds  to,  and  reports  security  incidents  in  ac-  detected  and  declared  as  incidents? 
cordance  with  established  standards  and  guidelines. 


Table  6:  Goals  to  Questions — Forbes  Scenario 


Goal 


Question 


G1 :  Ensure  users  whose  accounts  are  compromised  do  Q1 :  What  is  the  process  for  identifying  recurring  com- 

not  succumb  to  the  same  attack(s)  again  (using  random  promised  accounts? 

testing  for  one  year  following  a  compromise). 


3.6  Topic  6:  Questions  to  Indicators 

To  derive  indicators  from  questions,  it  is  useful  to  answer  the  following  questions: 

•  What  data  (and  sometimes  in  what  form)  do  I  need  to  answer  the  question? 

•  Which  data  is  most  important? 

•  If  I  have  this  data,  will  I  be  able  to  answer  some  aspect  of  this  question? 


Table  7,  Table  8,  and  Table  9  provide  example  indicators  for  selected  questions  by  scenario. 


Table  7:  Questions  to  Indicators — Ensuring  Healthy  Teeth 

Question 

Indicator 

G1  ,Q2:  Does  the  child  know  how  to  brush  properly? 

G2.Q1 :  Does  the  child  show  you  his/her  clean  teeth? 

Q2.I1:  Demonstration  of  use 

Q2.I2:  Issues  found  during  dental  checkups 

Q1 .11 :  Evidence  that  tooth  brushing  has  occurred 

Table  8:  Questions  to  Indicators — Incident  Management 

Question 

Indicator 

Q1 :  What  is  the  process  by  which  suspicious  events  are 
detected  and  declared  as  incidents? 

Q1.I1:  Process  and  criteria  for  detecting  and  triaging 
suspicious  events 

Table  9:  Questions  to  Indicators — Forbes  Scenario 

Question 

Indicator 

Q1 :  What  is  the  process  for  identifying  recurring  com¬ 
promised  accounts? 

Q1.I2:  Security  incident  reports  in  which  the  incident  is 
caused  by  the  same  user  account 

3.7  Topic  7:  Indicators  to  Metrics 

To  derive  metrics  from  indicators,  it  is  useful  to  answer  the  following  questions: 
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•  Using  the  indicator  data,  what  number,  percentage,  mean,  or  other  metric  can  I  collect  or  cal¬ 
culate  to  help  answer  the  question? 

•  Which  metrics  are  most  important? 

•  If  I  report  this  metric  (over  time),  will  it  provide  the  greatest  insight  possible  to  answer  the 
questions  from  which  it  derives? 

To  further  define  appropriate  metrics,  it  is  useful  to  answer  the  following  questions: 

•  Who  is  the  metric  for?  Who  are  the  stakeholders?  Who  collects  the  measurement  data? 

•  What  is  being  measured? 

•  Where  is  the  data/information  stored? 

•  When/how  frequently  are  the  metrics  collected? 

•  Why  is  the  metric  important  (vs.  others)? 

•  How  is  the  data  collected?  How  is  the  metric  presented?  How  is  the  metric  used? 


Table  10,  Table  11,  and  Table  12  provide  example  metrics  using  selected  indicators  by  scenario. 
Table  10:  Indicators  to  Metrics — Ensuring  Healthy  Teeth 


Indicator 

Metric 

Q2.I2:  Issues  found  during  dental  checkups 

Q1 .11 :  Evidence  that  tooth  brushing  has  occurred 

12. Ml:  Number  of  cavities 

12. M2:  Instances  of  gingivitis 

11. Ml:  Smell  of  breath 

11. M2:  Condition  of  toothbrush  (wet  vs.  dry) 

Table  1 1:  Indicators  to  Metrics — Incident  Management 

Indicator 

Metric 

Q1.I1:  Process  and  criteria  for  detecting  and  triaging 
suspicious  events 

Q1.I1.M1:  Mean  time  to  detect  suspicious  events 

Table  12:  Indicators  to  Metrics — Forbes  Scenario 

Indicator 

Metric 

Q1.I2:  Security  incident  reports  in  which  the  incident  is 
caused  by  the  same  user  account 

12. Ml :  Number  of  user  accounts  that  have  been  com¬ 
promised  by  the  same  attack 

12. M2:  Mean  time  between  similar  attacks  for  a  given 
user  account 

In  an  effective  measurement  program,  metrics  are  collected,  interpreted,  refined,  and  unproved  on 
an  ongoing  basis.  At  the  completion  of  the  GQIM  process,  there  is  an  initial  set  of  metrics  that  can 
be  used  to  monitor  the  organization’s  business  objectives.  However,  metrics  need  to  be  routinely 
revisited  (through  the  GQIM  process  or  another  approach)  to  ensure  decision  makers  are  receiving 
the  most  useful  and  actionable  metrics  to  monitor  progress  against  enterprise  objectives. 

3.8  Topic  8:  The  Big  Picture 

The  final  section  of  the  workshop  focused  on  understanding  the  “So  what?”  of  the  GQIM  process. 
We  asked  our  participants  to  consider  the  following  questions  to  determine  if  they  had  identified 
meaningful  metrics: 

•  What  decision  will  the  metric  infonn? 
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•  What  actions  would  I  take  based  on  this  metric? 

•  What  behaviors  would  the  metric  affect? 

•  What  would  improvements  look  like? 

Participants  were  encouraged  to  refine  their  metrics  as  they  continue  to  improve  and  develop  their 
measurement  programs.  The  GQIM  process  is  not  meant  to  be  only  executed  at  program  creation; 
rather,  it  can  and  should  be  used  to  routinely  revisit  and  improve  metrics  as  part  of  a  risk  manage¬ 
ment  approach.  Each  step  in  the  process  and  the  corresponding  questions  driving  each  step  are 
part  of  a  toolkit  that  should  be  used  to  continuously  improve  an  organization’s  measurement  pro¬ 
gram. 

During  the  closing  of  the  workshop,  we  revisited  participant  expectations  and  challenges.  We 
asked  participants  to  identify  approaches  to  take  back  to  their  organizations  to  address  current 
challenges  and  barriers  they  faced  in  developing  a  measurement  program  and  identifying  mean¬ 
ingful  metrics.  The  following  approaches  were  identified: 

•  Tie  metrics  to  business  objectives  and  put  the  outputs  in  the  language  of  the  business. 

•  Increase  awareness  of  the  purpose  of  measurements  and  how  to  tie  them  to  business  objec¬ 
tives. 

•  Make  sure  you  are  asking  the  right  questions. 

•  Focus  on  the  potential  impact  to  show  the  value  of  a  measurement  program  instead  of  trying 
to  prove  what  was  prevented. 

•  Things  are  not  going  to  get  less  complex,  so  make  sure  you  are  asking  the  right  questions 
and  measuring  the  right  things.  Also,  start  with  measuring  one  thing;  don't  try  to  boil  the 
ocean. 

•  Train  users  and  leadership  on  the  GQIM  process  and  encourage  them  to  use  it. 
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4  Workshop  Feedback 

Participants  provided  the  following  information  in  response  to  our  request  for  strengths  and  areas 
requiring  improvement  on  the  workshop  evaluation  form: 

Strengths 

Areas  for  Improvement 

Workshop  pre-work  (objectives,  Forbes  scenario) 

Provide  more  time  for  general  conversation  and 
knowledge  sharing  rather  than  on  the  table  exercises 

Having  a  well-defined,  structured  approach  for  deriving 
metrics  from  business  objectives 

Suggest  starting  out  with  examples  of  a  set  of  metrics 
that  have  worked  in  changing  behaviors.  Then,  back  up 
and  go  through  the  process  of  getting  there. 

Applicability  to  C-level  perspectives;  ability  to  use  ap¬ 
proach  with  business  stakeholders 

Spend  more  time  identifying  and  describing  the  right 
objectives  and  goals 

Instructor  engagement  during  group  exercise 

Provide  more  work  time  during  group  exercises 

Demonstration  of  how  to  develop  good  objectives 

Provide  more  guidance  on  how  to  move  from  questions 
to  metrics 

GQIM  examples  (brushing  teeth,  incident  management, 
Forbes,  several  table  topics) 

Provide  examples  of  metrics  that  have  worked 

Hands-on  approach 

Spend  more  time  on  metrics,  assessing  what  is  valua¬ 
ble  to  measure  vs.  what  is  not;  demonstrate  the  use  of 
the  metrics  template 

Small  group  work  and  discussions;  opportunity  to  col¬ 
laborate  with  peers 

Provide  time  during  group  work  for  participants  to  share 
metrics  that  they  are  reporting  and  how  they  are  doing  it 

Group  exercises  and  the  way  they  built  upon  each 
other;  the  workshop  format  was  easy  to  follow 

More  interaction  with  instructors  during  group  exercises 

Group  topics  were  relevant  and  real  life 

Expand  this  to  a  2-day  course 

Relevant  to  my  job 

Multiple  instructors;  well  prepared;  instructor  knowledge 
and  experience;  team  teaching  approach;  variety  of 
presentation  styles;  coordination  between  instructors 
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5  Next  Steps 


Going  forward,  the  SEI  will  offer  the  Measuring  What  Matters  Workshop  as  a  one-day  public 
course.  We  are  working  to  consolidate  feedback  from  all  of  the  participants  to  improve  the  prepa¬ 
ration,  execution,  and  follow-up  activities  around  the  workshop.  We  have  had  additional  requests 
for  private  offerings  of  the  workshop  from  multiple  industry  partners.  In  addition  to  the  release  of 
this  report,  we  will  release  a  podcast  on  the  workshop  and  its  results. 
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